This approach looks simple, and I like the syntax. I am not familiar with meta path, though. Is this a standard, supported mechanism?

@tlambert03 may also be interested.

Can you please make a PR that doesn't take out all the old code, but adds this as a new function? That'd make the diff easier to review.

I had never heard of sys.meta_path either until today, but here it is on the docs: https://docs.python.org/3/library/sys.html#sys.meta_path

According to PEP 451, which introduced ModuleSpecs in Python 3.4, it seems sys.meta_path was introduced with PEP 302 for Python 2.3 in 2002.

Can you please make a PR that doesn't take out all the old code, but adds this as a new function? That'd make the diff easier to review.

Done!

*Well, more or less. If you want to keep the old code, I should also leave the previous tests.

I like the syntax too! but are you sure it's lazy?

I just checked out this PR, installed it, and created the following test case:

mod/
├── __init__.py
├── __init__.pyi
└── _mod.py

where _mod.py contains

print("IMPORT")

def some_func(x: int) -> int:
    return x + 1

lazy_loader.attach

# mod/__init__.py
import lazy_loader

__getattr__, __dir__, __all__ = lazy_loader.attach(
    __name__, submod_attrs={"_mod": ["some_func"]}
)

In [1]: import mod

In [2]: dir(mod)
Out[2]: ['some_func']

In [3]: mod.some_func
IMPORT
Out[3]: <function some_func at 0x1038af130>

lazy_loader.attach_stub

# mod/__init__.py
import lazy_loader

__getattr__, __dir__, __all__ = lazy_loader.attach_stub(__name__, __file__)

In [1]: import mod

In [2]: dir(mod)
Out[2]: ['some_func']

In [3]: mod.some_func
IMPORT
Out[3]: <function some_func at 0x107fbbeb0>

lazy_loader.lazy_import

# mod/__init__.py
import lazy_loader

with lazy_loader.lazy_import():
    from ._mod import some_func

‼️ greedy import

In [1]: import mod
IMPORT

In [2]: dir(mod)
Out[2]:
[
    '__builtins__',
    '__cached__',
    '__doc__',
    '__file__',
    '__loader__',
    '__name__',
    '__package__',
    '__path__',
    '__spec__',
    '_mod',
    'lazy_loader',
    'some_func'
]

I do think that adding a MetaPathFinder to sys.meta_path would potentially be another potentially elegant way to solve this, but i'm not sure this is doing what we want it to?

it eagerly raises an ImportError if a module is not found,

I think this is because it's actually eagerly importing?

But from ._mod import some_func is accesing an attribute of ._mod, hence it imports _mod.

You need to do:

import lazy_loader

with lazy_loader.lazy_import():
    import mod  # lazy

mod.some_func  # triggers import

as in the examples where you used lazy_loader.attach.

it eagerly raises an ImportError if a module is not found,

I think this is because it's actually eagerly importing?

The import system seems to be divided in finding and loading. As it doesn't find the module, it can raise an eager ImportError. But if it is found, it isn't yet loaded, waiting until first attribute access..

I'm a bit confused... in the current implement of lazy.attach, the goal is to be able to do something like:

>>> import mod   # mod._mod not yet imported
>>> mod.some_func()  # on attribute access, NOW mod._mod is imported and some_func taken

or ... another example taken from SPEC 1:

import skimage as ski  # cheap operation; does not load submodules

ski.filters  # cheap operation; loads the filters submodule, but not
             # any of its submodules or functions

ski.filters.gaussian(...)  # loads the file in which gaussian is implemented
                           # and calls that function

are you suggesting that it would be mod.mod.some_func? and doesn't changing the test here to assert isinstance(fake_pkg.some_func, types.ModuleType) suggest that you've changed what SPEC 1 is trying to achieve?

and in your example, how would you declare what the public exports are?

Ok, now I understand that that test.

Yes, you're right. What I did is different to SPEC 1, and only works for modules, but not for exported functions.

For instance, if skimage.__init__.py were:

from lazy_loader import lazy_import

with lazy_import():
    from . import filters, morphology, ...

it would lazily export all those subpackages. And then, you could do:

import skimage as ski  # cheap; does not load submodules

skimage.filters  # only loads the filters submodule

where skimage.filters is lazily loaded, and the same is valid for any submodule inside skimage.filters (which is also inside a with lazy_import(), of course).

But skimage.filters.__init__.py only exports a single module, rank and then exports all functions, which are equivalent to from .submodule import function and would be eagerly loaded by what I propose.

So, in conclusion, it does not replace attach. But it could take over the submodules parameter of attach, wouldn't it?

So, in conclusion, it does not replace attach. But it could take over the submodules parameter of attach, wouldn't it?

yep, that jives with my understanding of what it's doing

Thanks all, that's a helpful discussion. I'm going to close this for now, as it does not cover the intent of the SPEC, but of course this remains a cool trick that libraries could use internally. Thanks for participating, @maurosilber, much appreciated!